Skip to content

feat(digest): port Digest::SHA3 via Bouncy Castle; refactor NetSSLeay key parsing#522

Merged
fglock merged 3 commits intomasterfrom
feature/digest-sha3-bouncycastle
Apr 21, 2026
Merged

feat(digest): port Digest::SHA3 via Bouncy Castle; refactor NetSSLeay key parsing#522
fglock merged 3 commits intomasterfrom
feature/digest-sha3-bouncycastle

Conversation

@fglock
Copy link
Copy Markdown
Owner

@fglock fglock commented Apr 21, 2026

Summary

Ports Digest::SHA3 to PerlOnJava via Bouncy Castle, and uses the
newly-added dependency to clean up private-key DER parsing in
NetSSLeay.java. See dev/modules/digest_sha3.md for the full
design rationale.

Before this change, ./jcpan -t Digest::SHA3 installed the CPAN
distribution but failed 11/14 tests with Undefined subroutine &Digest::SHA3::newSHA3 etc., because Digest::SHA3 is an XS module
and PerlOnJava had no Java-side port.

What's in this PR

  1. Dependency: adds bcprov-jdk18on + bcpkix-jdk18on v1.78.1
    in gradle/libs.versions.toml and build.gradle.

  2. DigestSHA3.java (new): BC-backed Keccak wrapper
    implementing the full XS interface of CPAN Digest-SHA3-1.05
    (newSHA3, shainit, sharewind, shawrite, add, digest,
    hexdigest, b64digest, squeeze, clone, hashsize,
    algorithm, and the 18 one-shot functions from sha3_224 through
    shake256_base64).

    • Subclasses KeccakDigest directly and applies the SHA-3 ("01")
      and SHAKE ("1111") domain separators in Java — SHA3Digest.doFinal
      and SHAKEDigest.doOutput call absorbBits internally, which
      fails after we've already flushed a non-byte-aligned reservoir.
    • Bit reservoir supports multi-call non-byte-aligned add_bits —
      CPAN bitorder.t does ->add_bits("0")->add_bits("1")->add_bits("1")
      which BC cannot handle through absorbBits alone.
    • No Perl shim needed: the unmodified CPAN lib/Digest/SHA3.pm
      calls XSLoader::load('Digest::SHA3') which dispatches to our
      Java class via XSLoader.java.

  3. NetSSLeay.parsePrivateKeyDer refactor: replaces a
    trial-and-error loop over {RSA, EC, DSA, EdDSA} KeyFactory
    calls plus a hand-rolled PKCS#1→PKCS#8 DER wrapper with
    PrivateKeyInfo.getInstance + JcaPEMKeyConverter. Auto-detects
    the algorithm from the DER AlgorithmIdentifier and now supports
    Ed25519/Ed448 keys for free. Deletes wrapPkcs1InPkcs8 and its
    local DER helpers.

  4. Doc updates:

    • dev/modules/digest_sha3.md (new): design + implementation log.
    • dev/modules/netssleay_complete.md: resolves the "adopt Bouncy
      Castle?" open question and updates the runtime-dependencies
      section.

Explicitly out of scope (future PRs)

  • Encrypted-PEM write path with traditional SSLeay format (stubbed today).
  • DH parameter PEM parsing.
  • PKCS#12 with non-standard MACs.
  • CSR builder / X509 extension DER rewrite via
    JcaPKCS10CertificationRequestBuilder and org.bouncycastle.asn1.*.
    The remaining hand-rolled DER code is extensively used and fully
    tested; a mechanical BC port deserves its own focused PR.
  • Fixing Digest::SHA.add_bits partial-byte handling (BC doesn't
    expose bit-level SHA-2 primitives; would need reflection or a
    hand-rolled compression loop).
  • New CPAN crypto ports now unblocked: Digest::Keccak,
    Digest::BLAKE2, Digest::BLAKE3, Digest::HMAC_SHA3_*,
    Crypt::OpenSSL::AES, Crypt::CBC backends, Crypt::JWT.

Test plan

  • ./jcpan -t Digest::SHA3 → 14/14 test files pass (33 subtests),
    including t/bit-sha3-{224,256,384,512}.t,
    t/bit-shake{128,256}.t, t/bitorder.t, t/sha3-{224,256,384,512}.t.
  • prove -e ./jperl src/test/resources/unit/netssleay_*.t →
    2553/2553 tests pass (no regressions in the BC-based
    parsePrivateKeyDer).
  • make → all unit tests green.

Size impact

  • +8 MB for bcprov-jdk18on, +3 MB for bcpkix-jdk18on. Modest
    compared to the existing icu4j (~12 MB).
  • Runtime dependency, MIT-style license (compatible with Apache 2.0).
  • Not a FIPS build — if FIPS is ever required, switch to bc-fips.

Generated with Devin

@fglock fglock force-pushed the feature/digest-sha3-bouncycastle branch from 2a6fafe to dc78c62 Compare April 21, 2026 10:39
fglock and others added 2 commits April 21, 2026 13:13
@fglock @devin-ai-integration
feat(digest): port Digest::SHA3 via Bouncy Castle; refactor NetSSLeay…
7d1affd 
… key parsing

Before this change, `./jcpan -t Digest::SHA3` installed the CPAN module
but 11 of 14 tests failed with "Undefined subroutine &Digest::SHA3::*"
because Digest::SHA3 is an XS module with no PerlOnJava backend.

This PR:

1. Adds Bouncy Castle (bcprov-jdk18on + bcpkix-jdk18on 1.78.1) as a
   mandatory runtime dependency. See dev/modules/digest_sha3.md for the
   analysis of why BC is the right choice (bit-level input + SHAKE XOFs
   + state copy, none of which the JDK provides for SHA-3).

2. Adds src/main/java/org/perlonjava/runtime/perlmodule/DigestSHA3.java
   — a BC-backed Keccak wrapper implementing the full XS interface of
   CPAN Digest-SHA3-1.05 (newSHA3, shainit, sharewind, shawrite, add,
   digest, hexdigest, b64digest, squeeze, clone, hashsize, algorithm,
   and the 18 one-shot functions sha3_224..shake256_base64).

   The wrapper subclasses KeccakDigest directly and applies the SHA-3
   ("01") and SHAKE ("1111") domain separators in Java, because
   SHA3Digest.doFinal and SHAKEDigest.doOutput call absorbBits
   internally, which fails after we've already flushed a non-byte-
   aligned reservoir.

   A bit reservoir supports multi-call non-byte-aligned add_bits() —
   the CPAN `bitorder.t` test does e.g. `->add_bits("0")->add_bits("1")
   ->add_bits("1")` which BC cannot handle directly.

3. Refactors NetSSLeay.parsePrivateKeyDer to use BC's PrivateKeyInfo +
   JcaPEMKeyConverter. Replaces a trial-and-error loop over {RSA, EC,
   DSA, EdDSA} KeyFactory.generatePrivate calls plus a hand-rolled
   PKCS#1→PKCS#8 DER wrapper (wrapPkcs1InPkcs8, now deleted). The
   AlgorithmIdentifier auto-detection in PrivateKeyInfo is correct by
   construction and now supports Ed25519/Ed448 keys for free.

4. Resolves the "adopt Bouncy Castle?" open question in
   dev/modules/netssleay_complete.md. The remaining hand-rolled DER
   code (CSR builder, X509 extension encoders, RDNs, SAN encoding)
   stays for this PR — it's extensively used and has full test
   coverage; incremental BC ports are follow-up work.

Test results:

- `./jcpan -t Digest::SHA3`: 14/14 test files pass (33 subtests),
  including all bit-level and SHAKE tests.
- `prove -e ./jperl src/test/resources/unit/netssleay_*.t`:
  2553/2553 tests pass (no regressions in private-key parsing).
- `make`: all unit tests green.

See dev/modules/digest_sha3.md for the full design rationale and the
per-step implementation log.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock @devin-ai-integration
fix(NetSSLeay): avoid double-invoking passwd callback in CTX_use_Priv…
56c79b7 
…ateKey_file

CTX_use_PrivateKey_file previously called loadPrivateKeyFile (which
invokes the password callback) and then re-opened the PEM and
re-invoked the callback to populate ctxState.loadedPrivateKey for
buildSslContext. This broke t/local/05_passwd_cb.t, which counts
callback invocations:

  not ok 17 - different cbs per ctx work    # each cb called 2x, expected 1
  not ok 21 - callback1 called 2 times      # got: '3' expected: '2'

Plus spurious "Bad plan: planned 36 tests but ran 40" because the
extra callback calls ran their own is()/ok() assertions.

Fix: loadPrivateKeyFile now takes an optional SslCtxState and
populates loadedPrivateKey + clears the cached SSLContext in a single
pass. The callback runs exactly once per load. use_PrivateKey_file
(SSL-level) benefits too — it now also populates the underlying
CTX's key so the KeyManager sees it.

Test results on t/local/05_passwd_cb.t:

  before: Tests: 40 Failed: 6  Parse errors: planned 36 ran 40
  after:  Tests: 36 Failed: 0  All tests successful.

Full Net-SSLeay bundled suite: 47 files, 2326 tests, all pass.
./jcpan -t Digest::SHA3: 33/33 still green. make: green.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock fglock force-pushed the feature/digest-sha3-bouncycastle branch from cefa787 to 56c79b7 Compare April 21, 2026 11:14
@fglock @devin-ai-integration
fix(ci): add Bouncy Castle dependencies to pom.xml
534812d 
The Gradle build (`make`, `make dev`) works locally because
`build.gradle` + `libs.versions.toml` were updated in the initial
commit. But the CI workflow uses Maven (`make ci` → `mvn`), and
`pom.xml` was not updated, so CI failed with

  package org.bouncycastle.crypto.digests does not exist
  package org.bouncycastle.asn1.pkcs does not exist
  ...

Add `bcprov-jdk18on` + `bcpkix-jdk18on` 1.78.1 to `pom.xml` to match
`libs.versions.toml`. Verified with `mvn -q compile`.

Generated with [Devin](https://cli.devin.ai/docs)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@fglock fglock merged commit 73edc8a into master Apr 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@fglock